Determining context to initiate interactivity

ABSTRACT

Methods and systems are disclosed for executing a voice command based on and association of the voice command and one or more identifiers. Audio data associated with a content asset may be received at a user device such as a voice activated device. A voice command may also be received at the user device. One or more identifiers associated with the audio data, such as a content or product identifier, may be determined. The identifiers may be determined based on playback of the content asset or may be received in response to a request generated by the user device. One or more operations capable of being executed by the user device may be determined and initiated or executed by the user device based on the one or more identifiers and the received voice command.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/880,448, filed Aug. 3, 2022, which is a continuation of U.S. patentapplication Ser. No. 15/791,936, filed Oct. 24, 2017, now U.S. Pat. No.11,445,235, issued Sep. 13, 2022, the entireties of which areincorporated herein by reference.

BACKGROUND

Voice activated devices may be controlled using audio data such as vocalinstructions or utterances from a user. A given voice activated devicemay be configured to receive an audio signal comprising a trigger and avoice command, and verification of the trigger may cause the voiceactivated device to output a response based on the received voicecommand. However, it may be desirable to recognize context andenvironmental factors associated with a voice command in order toenhance the accuracy of the voice activated device in responding to thevoice command. These and other shortcomings are addressed in thedisclosure.

SUMMARY

In one aspect, methods and systems are disclosed for executing a voicecommand based on one or more identifiers. Audio data associated with acontent asset may be received at a user device such as a voice activateddevice. A voice command that is separate from the audio data may also bereceived at the user device. One or more identifiers associated with theaudio data, such as a content identifier or an audio watermark, may bedetermined. The identifiers may be determined based on playback of thecontent asset or may be received at the user device from a playbackdevice. The identifiers may be sent to the user device in response to arequest generated by the user device, or may be pushed to the userdevice based on a predetermined conditions. One or more operationscapable of being executed by the user device may be determined andexecuted by the user device based on the one or more identifiers and thereceived voice command. These and other embodiments are furtherexplained below.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description is better understood when read inconjunction with the appended drawings. For the purposes ofillustration, examples are shown in the drawings; however, the subjectmatter is not limited to specific elements and instrumentalitiesdisclosed. In the drawings:

FIG. 1 is a block diagram of an example system;

FIG. 2 is a flow chart of an example method;

FIG. 3 is an example system operation according an aspect of thedisclosure;

FIG. 4 is a flow chart of an example method;

FIG. 5 is a flow chart of an example method;

FIG. 6 is an example system operation according an aspect of thedisclosure;

FIG. 7 is an example system operation according to an aspect of thedisclosure;

FIG. 8 is a flow chart of an example method; and

FIG. 9 is a block diagram of an example computing device.

DETAILED DESCRIPTION

Methods and systems are disclosed for determining one or more executableoperations based on a voice command and one or more identifiers. Userdevices such as voice activated devices may rely on the words or soundscontained in a voice command in determining and generating an output inresponse to the voice command. However, it may be possible to determinecontext associated with one or more other audio data in order to improvethe user experience. For example, the user device may be configured toreceive first audio data from a first source such as from a playbackdevice generating an audio output associated with playback of a contentasset, and to receive a voice command from a second source such as auser of the device. The audio data received at the user device maycomprise one or more identifiers. The user device may be configured todetermine, based on the one or more identifiers and the voice command,one or more operations that are capable of being executed by the deviceand to execute one or more of those operations.

The one or more identifiers may be received as audio data from a nearbydevice such as during playback of a content asset by a playback device.For example, a television asset may reference a product or a newlyreleased movie, each of which may be associated with an identifier suchas the name of the product or the newly released movie. The voicecommand may be received from a user of the device and may comprise oneor more keywords capable of being recognized by the device, such as“order,” “play,” and “record.” The one or more executable operations maybe determined based on the voice command and the one or moreidentifiers, and may include an operation to record a content assetassociated with a given identifier or to order a product associated witha given identifier through an online shopping account associated with auser.

An example system 100 for determining one or more executable operationsis shown in FIG. 1 . The system 100 may comprise a user device 102 and aplayback device 120. The user device 102 may be a voice activated deviceconfigured to receive audio data generated by a user or by a nearbydevice such as the playback device 120. The audio data may be receivedby the device as an audio input. For example, the user device 102 may beconfigured to receive at least one of audio data associated with acontent asset and a voice command spoken by a user of the device throughthe microphone 104. The audio data may be received, for example, duringplayback of the content asset by the playback device 120. The audio dataand the voice command may be received by the user device 102simultaneously or nearly simultaneously. In response to the audio dataand the voice command, the user device 102 may be configured todetermine one or more executable operations and to cause executions ofone or more of those operations. The user device 102 may be incommunication with one or more other devices such as the playback device120 in order to execute one or more voice commands spoken by a user andthat are capable of being executed by the playback device 120.

The user device 102 may be an “always-on” user device that is configuredto monitor for the presence of audio signals generated by nearby devicesor users of those devices. The always-on device may be configured tomonitor for one or more identifiers and one or more keywords that causethe always-on device to execute some operation associated with a voicecommand following the keyword. In one example, the voice command may bethe keyword itself and detection of the keyword by the user device 102may cause the user device 102 to execute an operation associated withthe keyword.

Additionally or alternatively, the user device 102 may be configured toenter a “sleep” mode after a certain period of time and may not monitorfor one or more identifiers from nearby devices or users of thosedevices. During a sleep mode, the user device 102 may be configured toignore all audio data until a keyword or other trigger is detected atthe user device 102. Upon the detection of one or more recognizedkeywords, the user device 102 may be configured to receive one or moreidentifiers associated with the audio signal from an external sourcesuch as the playback device 120. The user device 102 may also beconfigured to interact with a user of the device through the speaker106, such as by generating an output confirming the execution of a voicecommand uttered by a user of the device.

The identifier storage module 108 may be configured to store one or moreidentifiers associated with the audio signal. In the example that theuser device 102 is an “always-on” device, the user device 102 may beconfigured to store one or more identifiers received as part of audiodata during playback of a content asset. For example, the one or moreidentifiers may identify the particular content asset or one or moreproducts presented during playback of the content asset. The user device102 may be configured to store a plurality of identifiers associatedwith a given content asset. In one example, the user device 102 may beconfigured to store a plurality of identifiers associated with a contentasset for a limited period of time, such as thirty seconds. Thus, anidentifier received during playback of the content asset may be storedby the user device 102 in the identifier storage module 108 for thirtyseconds before being deleted by the user device 102.

The speech processor 110 may be configured to recognize one or morewords or phrases received at the user device 102 as audio data or avoice command. The speech processor 110 may comprise, for example, oneor more of a speech capture module, a digital signal processor (DSP)module, a preprocessed signal storage module, and a reference speechpattern and pattern matching algorithm module. Speech processing may bedone in a variety of ways and at different levels of complexity, forexample, using one or more of pattern matching, pattern and featureanalysis, and language modeling and statistical analysis. However, it isunderstood that any type of speech processing may be used, and theexamples provided herein are not intended to limit the capabilities ofthe speech processor 110.

The speech processor may be configured with a number of keywords, suchas those stored in keyword storage module 112. The keyword storagemodule 112 may comprise a number of keywords that are recognizable bythe speech processor 110 using, for example, pattern matching, and mayallow a user to interact with content presented during playback of acontent asset. For example, the keyword storage module 112 may store aplurality of commonly spoken words for interacting with the content,such as “order,” “play,” and “record.” Thus, when audio data generatedas a result of playback of a content asset comprises a reference to aproduct, and the user device 102 receives a voice command comprising thekeyword “order,” the user device 102 may be configured to order theproduct, based on the identifier associated with the product, through anonline shopping account associated with the user. Additionally oralternatively, when audio data generated as a result of playback of acontent asset comprises a reference to an upcoming television series,and the user device 102 receives a voice command comprising the keyword“record,” the user device 102 may be configured to instruct the playbackdevice 120 to record the upcoming television series using the identifierassociated with the upcoming television series.

The playback device 120 may comprise a playback module 122 and a userinterface module 124. The playback device 120 may be configured toreceive a request from a user or from a nearby device such as the userdevice 102 to record or play back a content asset. The playback device120 may be, for example, a set-top box or a cellular telephone. Theplayback device 120 may be configured to present a content asset to auser over one or more screens or monitors. The content asset may be anytype of video asset, for example, a video clip, a movie asset, atelevision asset, a live broadcast asset such as a sporting event, anaudio as set such as music streaming, or a Video on Demand (VOD) asset.However, it is understood that the content asset may be any type ofcontent asset having any length or duration. The content asset may be inany one of a variety of formats, such as, for example, H.264, MPEG-4Part 2, or MPEG-2. The content asset may be transmitted using one ormore standards such as SCTE 35 or other specifications.

Digital audio/video compression may be utilized for the content asset,such as MPEG, or any other type of compression. For example, in an MPEGencoded transmission, content and other data are transmitted in packets,which collectively make up a transport stream. However, the presentmethods and systems are not so limited, and may be implemented usingother types of transmission and data.

The output of a single MPEG audio and/or video coder may be referred toas a transport stream. The transport stream may comprise one or moreelementary streams. An elementary stream may be or comprise an endlessnear real-time signal. For convenience, the elementary stream may bebroken into data blocks of manageable size, forming a packetizedelementary stream (PES). These data blocks need header information toidentify the start of the packets and must include time stamps becausepacketizing disrupts the time axis. For transmission and digitalbroadcasting, for example, several programs (e.g., content assets) andtheir associated PESs may be multiplexed into a multi program transportstream. A multi program transport stream has a program clock reference(PCR) mechanism that allows transmission of multiple clocks, one ofwhich may be selected and regenerated at the decoder.

A multi program transport stream may comprise a multiplex of audio andvideo PESs. In addition to the compressed audio, video and data, atransport stream may comprise metadata describing the bit stream. Suchmetadata may comprise a program association table (PAT) that lists everycontent asset (e.g., program) in the multi program transport stream.Each entry in the PAT may point to a program map table (PMT) that liststhe elementary streams making up each content asset. Some content may beunencrypted, but some content may be subject to conditional access(encryption) and this information is also carried in the metadata. Thetransport stream may be comprised of fixed-size data packets, forexample, each containing 188 bytes. Each packet may carry a programidentifier code (PID). Packets in the same elementary stream may allhave the same PID, so that the decoder (or a demultiplexer) may selectthe elementary stream(s) it wants and reject the remainder. Packetcontinuity counts ensure that every packet that is needed to decode astream is received. A synchronization system may be used so thatdecoders may correctly identify the beginning of each packet anddeserialize the bit stream into words.

A content asset, such as a program, may be a group of one or more PIDsthat are related to each other. For instance, a multi program transportstream used in digital television might contain three programs, torepresent three television channels. In some example, each channel mayinclude one video stream, one or two audio streams, and any necessarymetadata. A receiver wishing to tune to a particular “channel” merelyhas to decode the payload of the PIDs associated with its program.

As discussed herein, the content asset may comprise one or moreidentifiers. The one or more identifiers may be inserted into thecontent asset in order to provide context for one or more features ofthe content asset. The one or more identifiers may be inserted into thecontent asset, for example, by the content provider, the set-top box, orby any device or entity capable of inserting the identifiers into thecontent asset. Additionally or alternatively, the user device 102 may beconfigured to perform content recognition processing, such as automaticcontent recognition (ACR), in order to determine context associated withthe content asset.

The playback module 122 may be in communication with an external serviceprovider such as a content provider in order to present playback of thecontent asset to the user. The playback module 122 may be incommunication with a device configured to present the content asset tothe user. For example, the playback module 122 may be electricallyconnected to a television set or may be connected to a hand-held devicesuch as a cellular telephone through a WiFi connection in order topresent the content to the user. As the content is presented on the oneor more devices configured to present the content to a user, audio datamay be generated that contains one or more identifiers inserted into thecontent asset. The audio data may be generated in the form of a soundwave that is capable of being picked up by one or more microphones. Forexample, the audio data and the corresponding one or more identifierspresent in the audio data generated as a result of playback of thecontent asset may be received by the user device 102 as audio data.

The identifier storage module 124 may be configured to store one or moreidentifiers associated with playback of a content asset. The identifierstorage module 124 may be configured to receive the identifiers from theservice provider. Additionally or alternatively, the identifier storagemodule may be configured to determine one or more identifiers generatedas a result of playback of the content asset, for example, using speechprocessing or an existing transcription of the content asset. Theidentifier storage module 124 may be configured to store the one or moreidentifiers and to send the one or more identifiers to the user device102 in response to a request from the user device.

The computing device 130 may comprise a speech processor 132 and anidentifier storage module 134. The computing device 130 may beconfigured to receive, from the user device 102, audio data such as avoice command and may generate a transcription of the voice commandusing the speech processor 132. Additionally or alternatively, thecomputing device may be configured to determine, using the identifierstorage module 134, an identifier associated with the voice commandbased on the transcription of the voice command generated by the speechprocessor 132.

The speech processor 132 may be configured to recognize an audio filecomprising one or more words received spoken by a user of the voiceactivated device and to output a transcription of the audio file. Thespeech processor 132 may comprise, for example, one or more of a speechcapture module, a digital signal processor (DSP) module, a preprocessedsignal storage module, and a reference speech pattern and patternmatching algorithm module. Speech recognition may be done in a varietyof ways and at different levels of complexity, for example, using one ormore of pattern matching, pattern and feature analysis, and languagemodeling and statistical analysis. However, it is understood that anytype of speech recognition may be used, and the examples provided hereinare not intended to limit the capabilities of the speech processor 132.

The identifier storage module 134 may be configured to store one or moreidentifiers associated with playback of a content asset. The identifierstorage module 134 may be configured to receive the identifiers from theservice provider. Additionally or alternatively, the identifier storagemodule may be configured to determine one or more identifiers generatedas a result of playback of the content asset, for example, using speechprocessing or an existing transcription of the content asset. Theidentifier storage module 134 may be configured to store the one or moreidentifiers and to send the one or more identifiers to the user device102 in response to a request from the user device.

FIG. 2 illustrates an example method 200 in accordance with an aspect ofthe disclosure. At step 202, audio data may be received. The audio datamay be associated with a content asset, such as a movie, television, oraudio asset. The audio data may be received at the device from a firstentity, such as the playback device 120 (FIG. 1 ) configured to causeplayback of the content asset. The audio data may be received at theuser device 102 (FIG. 1 ) which may be a voice activated device. Theuser device may be an always-on device that is configured to monitor forthe presence audio signals generated by nearby devices (e.g., theplayback device) or users of those devices. Additionally oralternatively, the user device may be configured to enter a “sleep” modeafter a certain period of time and may not “wake” until one or morekeywords are detected at the user device.

At step 204, a voice command may be received at the user device. Thevoice command may be received at the device from a second entity. Thesecond entity may be different from the first entity. For example, thesecond entity may be a user of the device uttering the voice command.The voice command may comprise one or more keywords capable of beingrecognized by the device. The user device may be configured to monitorfor one or more keywords that cause the always-on device to execute someoperation associated with the voice command. In the example that theuser device enters a sleep mode, the device may be configured to wakeupon the detection of the one or more keywords. The one or more keywordsmay be stored by the user device in the keyword storage module which maycomprise a number of keywords that are recognizable by the speechprocessor using, for example, pattern matching, and may allow a user tointeract with content presented during playback of a content asset. Forexample, the keyword storage module may store a plurality of commonlyspoken words for interacting with the content, such as “order,” “play,”and “record.”

At step 206, one or more identifiers associated with the audio data maybe determined. The one or more identifiers may comprise, for example, atleast one of an SCTE-35 packet, a unique program identifier (UID), acontent identifier, or an audio watermark. In one example, the one ormore identifiers may be determined using audio content recognitionprocessing. The one or more identifiers may be received as audio dataduring playback of the content asset by the playback device, e.g., atelevision, smartphone, etc. For example, a video asset may comprise areference to a product or a newly released movie, each of which may beassociated with an identifier such as the name of the product or thenewly released movie. The user device may be configured to store one ormore identifiers received as part of audio data during playback of acontent asset. For example, the user device may be configured to receivethe identifiers from the source of the content or a service provider.Also, the user device may be configured to determine the one or moreidentifiers generated as a result of playback of the content asset, forexample, using speech processing or an existing transcription of thecontent asset.

The user device may be configured to store a plurality of identifiersfor a determined period of time. For example, the user device may beconfigured to receive an identifier and to store the identifier in theidentifier storage module for a limited period of time, such as thirtyseconds. In such an example, the user device may be configured to deleteany identifier that has been stored in the identifier storage module forgreater than thirty seconds. The user device may comprise a rollingbuffer that is configured to continuously record audio data includingthe one or more audio signals and to delete the recorded data and audiosignals after a determined period. Additionally or alternatively, theuser device may be configured to start recording the audio data as soonas it hears the trigger word and before the voice command is uttered bythe user.

At step 208, an association between the voice command and an identifiermay be determined. In one example, the user device may be configured tostore a plurality of identifiers, for example, in the identifier storagemodule of the user device and to determine one or more identifierscapable of being executed by the device. Determining one or moreoperations capable of being executed by the device may comprisedetermining that one or more of the identifiers associated with theaudio data were received within a given time frame of the voice command.For example, a user device may receive a voice command comprising theknown keyword “order.”

In one example, as shown in FIG. 3 , the user device may determine thatthe identifier storage module comprises three identifiers in the thirtyseconds prior to the voice command: a first identifier associated with afirst product and received twenty-five seconds before the voice command;a second identifier associated with a television asset received fifteenseconds before the voice command; and a third identifier associated witha second product received five seconds before the voice command. Theidentifiers may optionally be stored in the identifier storage module.In one embodiment, the identifier storage module may locally storecertain types of hierarchal identifiers. Additionally or alternatively,the identifier storage module may have access to a remote storage orprocessing device that can provide the identifiers. The user device maydetermine based on the known keyword “order” that the user wishes toplace an order for either the first product associated with the firstidentifier or the second product associated with the third identifier.The user device may further determine, since the third identifier wasreceived closer to the voice command than the first identifier, that theuser wishes to place an order for the second product associated with thethird identifier. While this example discloses three identifiersreceived at specified intervals, it is understood that any number ofidentifiers may be received over any number of intervals or timeperiods. For example, the user device may receive twenty identifiersover a two-minute period, and may use the timing of the receivedidentifiers as well as other characteristics of the identifiers in orderto determine context associated with the received voice command.

At step 210, execution of at least one or more of the determinedoperations may be initiated or executed. The user device may beconfigured to execute an operation determined based on the voice commandat the one or more identifiers stored in the user device. In the exampleabove, the user device may be configured to place an order for thesecond product associated with the third identifier through an onlineshopping account using stored information associated with the user. Theuser device may be configured to generate an audio output indicative ofthe one or more executable operations. For example, in response toplacing an order for the second product through the home shoppingnetwork, the user device may confirm that the order was placed bygenerating an audio output of “order for second product has beenplaced.” Additionally or alternatively, the user device may generate anaudio output confirming the voice command before executing the one ormore operations, such as by querying the user “are you sure you want toorder the second product?”

FIG. 4 shows an example system implementation according to an aspect ofthe disclosure. The playback device 120 may be configured to play back acontent asset, such as a movie, television or audio asset. The playbackdevice may be in communication with a service provider such as a contentor service provider and may in connection with a television or othermonitor in order to display the content asset to a user, such as user450. The user device 102, which may be a voice activated device, may beconfigured to monitor for the presence of audio signals in a nearbyarea, such as the audio signals corresponding to playback of the contentasset by the playback device 120. In this example, the user device maybe an always-on voice activated device that is configured to monitor forthe presence of audio signals generated by nearby devices or users ofthose devices. However, it is understood that the user device may be anytype of device. The audio signals received from the playback device maycomprise one or more identifiers corresponding, for example, to aparticular content asset or one or more products presented duringplayback of the content asset. The user device may be configured tostore the one or more identifiers in an identifier storage module of theuser device for a certain period of time, such as thirty seconds.

The user device may be configured to receive a voice command from theuser. The voice command may comprise one or more keywords that arecapable of being recognized by the user device using speech recognitionprocessing such as pattern matching. In response to receipt of the voicecommand, the user device may be configured to determine whether theidentifier storage module associated with the user device comprises oneor more identifiers. For example, the user device may receive a voicecommand comprising the keyword “record” and may determine that theidentifier storage module comprises one or more identifiers associatedwith content that is capable of being recorded. In response todetermining that the identifier storage module does comprise one or moreidentifiers associated with content that is capable of being recorded,the user device may instruct the playback device to record the contentassociated with the identifier. In one example, the user device may beconfigured to instruct one or more other playback devices capable ofrecording content to record the content associated with the identifier.

FIG. 5 illustrates an example method 500 in accordance with an aspect ofthe disclosure. At step 502, audio data may be received at a device,such as the user device 102. The audio data may be associated with acontent asset, such as a movie, television, or audio asset. The audiodata may be received at the device from a first entity, such as theplayback device 120. The audio data may be received at a user devicesuch as a voice activated device. The device may be an always-on devicethat is configured to monitor for the presence audio signals generatedby nearby devices (e.g., the playback device) or users of those devices.Additionally or alternatively, the user device may be configured toenter a “sleep” mode after a certain period of time and may not “wake”until one or more keywords are detected at the user device.

At step 504, a voice command may be received at the user device. Thevoice command may be received at the device from a second entity. Thesecond entity may be different from the first entity. For example, thesecond entity may be a user of the device uttering the voice command.The voice command may comprise one or more keywords capable of beingrecognized by the device. In the example that the device has entered asleep mode, the device may be configured to wake upon the detection ofthe one or more keywords. The one or more keywords may be stored by theuser device in the keyword storage module which may comprise a number ofkeywords that are recognizable by the speech processor using, forexample, pattern matching, and may allow a user to interact with contentpresented during playback of a content asset. For example, the keywordstorage module may store a plurality of commonly spoken words forinteracting with the content, such as “order,” “play,” and “record.”

At step 506, a request for one or more identifiers associated with theaudio data may be sent. The request for the one or more identifiers maybe sent in response to the receipt of a voice command at the userdevice. Thus, upon receiving a voice command with a keyword that iscapable of being recognized by the user device, the user device may beconfigured to “wake” from a sleep mode and to send a request for or moreidentifiers. The one or more identifiers may comprise, for example, atleast one of an SCTE-35 packet, a unique program identifier (UID), acontent identifier, or an audio watermark. Sending a request for one ormore identifiers associated with the audio data may comprise sending, tothe first entity, the request for the one or more identifiers associatedwith the audio data. However, the request for one or more identifiersmay be sent to any device capable of determining one or more identifiersassociated with playback of a content asset. The user device may beconfigured to determine, based on the voice command, a type ofidentifier to request from the first entity. Additionally oralternatively, the user device may be configured to send a request forany number of identifiers received within a certain time frame of thevoice command, such as a request for all identifiers received thirtyseconds prior to the voice command.

At step 508, the one or more identifiers may be received. The one ormore identifiers may be received at the user device from the playbackdevice that generated the audio data associated with playback of thecontent asset. For example, receiving the one or more identifiers maycomprise receiving the one or more identifiers from the first entity.However, it is understood that the one or more identifiers may bereceived from any device capable of recognizing or transmitting one ormore identifiers in a content asset. The user device may be configuredto store a plurality of identifiers associated with a content asset fora limited period of time, such as thirty seconds. The user device may beconfigured to receive one or more identifiers in response to a requestfor a particular identifier based on the voice command. Additionally oralternatively, the user device may receive one or more identifiers inresponse to sending a request for all identifiers within a given timeframe of the received voice command.

At step 510, an association between the voice command and an identifiermay be determined. In one example, the user device may be configured tostore a plurality of identifiers, for example, in the identifier storagemodule of the user device and to determine one or more identifierscapable of being executed by the device. Determining one or moreoperations capable of being executed by the device may comprisedetermining that one or more of the identifiers associated with theaudio data were received within a given time frame of the voice command.For example, a user device may receive a voice command comprising theknown keyword “order.”

Upon receiving the voice command, the user device may determine that theidentifier storage module has stored three identifiers in the thirtyseconds prior to the voice command: a first identifier associated with afirst product and received twenty-five seconds before the voice command;a second identifier associated with a television asset received fifteenseconds before the voice command; and a third identifier associated witha second product received five seconds before the voice command. Theuser device may determine based on the known keyword “order” that theuser wishes to place an order for either the first product associatedwith the first identifier or the second product associated with thethird identifier. The user device may further determine, since the thirdidentifier was received closer to the voice command than the firstidentifier, that the user wishes to place an order for the secondproduct associated with the third identifier.

At step 512, execution of at least one or more of the determinedoperations may be caused. The user device may be configured to executean operation determined based on the voice command and the one or moreidentifiers stored in the user device. In the example above, the userdevice may be configured to place an order for the second productassociated with the third identifier through an online shopping accountusing stored information associated with the user. The user device maybe configured to generate an audio output indicative of the one or moreexecutable operations. For example, in response to placing an order forthe second product through the home shopping network, the user devicemay confirm that the order was placed by generating an audio output of“order for second product has been placed.” Additionally oralternatively, the user device may generate an audio output confirmingthe voice command before executing the one or more operations, such asby querying the user “are you sure you want to order the secondproduct?”

FIG. 6 shows an example system implementation according to an aspect ofthe disclosure. The playback device 120 may be configured to play back acontent asset, such as a movie, television or audio asset. The playbackdevice 120 may be in communication with a service provider such as acontent or service provider and may be in connection with a televisionor other monitor in order to display the content asset to a user, suchas user 650. The user device 102, which may be a voice activated device,may be configured to enter a “sleep” mode after a given period of timeand until a voice command is detected at the user device 102. The voicecommand may comprise one or more keywords that are capable of beingrecognized by the user device 102 using speech recognition processingsuch as pattern matching. In response to receipt of the voice command,the user device 102 may be configured to send, to a device such as theplayback device 120, a request for an identifier associated with thecontent asset.

The playback device 120, in response to receiving the request for theone or more identifiers, may be configured to send to the user device102 at least one identifier. The playback device 120 may send to theuser device 102 one or more identifiers based on the received voicecommand or one or more identifiers detected within a given time frame ofthe received voice command. For example, the user device 102 may receivea voice command comprising the keyword “order.” The user device 102 maysend to the playback device 120 a request for one or more identifiersthat correspond to the keyword “order” or may send a request for one ormore identifiers received within a given time frame of the keyword“order.”

The playback device 120, in response to this request, may be configuredto send the one or more identifiers to the user device 102. In responseto receipt of the one or more identifiers, the user device 102 may beconfigured to execute an operation associated with the voice command.For example, the user device 102 may receive from the playback device120 one or more product identifiers present in a content asset, and maybe configured to place an order for one or more of the productsassociated with the identifiers by communicating with an online shoppingaccount over the network 630.

FIG. 7 shows an example system implementation according to an aspect ofthe disclosure. A user device, such as user device 102, may be incommunication with a control panel 720. The control panel 720 may beconfigured to control one or more of lighting, heating or airconditioning in a user's home. The user device 102 may be configured tocommunicate with the control panel 720 in order to execute voicecommands received from a user, such as to turn on or off one or morelights in the user's home or to adjust the temperature in the user'shome. For example, the user device 102 may receive from the user 750audio data comprising the voice command “turn off that light.” However,the user device 102 may not be able to determine from the voice commandalone which light in the home the user is referring to.

In response to receiving the voice command “turn off that light,” theuser device 102 may be configured to communicate with an externalnetwork 730. The network 730 may be able to determine the condition(e.g., on, off, dimmed, etc.) of one or more light switches in theuser's home, such as by communicating with a smart metering system inthe home. If the network 730 determines that there is only one light inthe user's home that is turned on, the network 730 may send to the userdevice 102 the identifier associated with that light. The user device102 may, in turn, execute an operation associated with the receivedvoice command by communicating with the control panel 720 to turn offthe light corresponding to the received identifier. Additionally oralternatively, the network 730 may determine that there is more than onelight on in the user's home and that more information is needed toexecute the voice command. In response to this determination, thenetwork 730 may send to the user device 102 a request for moreinformation. In response to receiving this request, the user device 102may query the user for more information, such as by generating an audiooutput “which light do you wish to turn off,” and may receive secondaudio data from the user, such as “kitchen light.” The user device 102may send this response to the network 730 along with a request for anidentifier associated with the kitchen light, and may receive theidentifier associated with the kitchen light. Upon receiving theidentifier, the user device 102 may execute the operation by sending tothe control panel 720 a request to turn off the light corresponding tothe identifier for the kitchen light.

Thus, the user device 102 may be configured to contact an externalnetwork such as a network 730 in order to determine context associatedwith a voice command. In another example, the user device 102 mayreceive first audio data such as playback of a content asset, and avoice command such as “show me movies with him in it.” The user device102 may send the first audio data to the network 730 where the network730 may be configured to perform audio content recognition (ACR)processing on the first audio data in order to determine at least one ofthe title of the movie or a list of actors in the movie. The network 730may send an identifier associated with the movie or an identifierassociated with at least one actor in the movie to at least one of theuser device 102 or the playback device 120. If there is more than oneactor in the movie, the user device 102 may be configured to query theuser in order to determine which actor the user is referring to. Theplayback device 120, upon receiving the identifier, may display via auser interface a list of movies with the actor corresponding to thereceived identifier.

FIG. 8 shows an example method according to an aspect of the disclosure.At step 802, a request for an identifier associated with audio data maybe received. The request for an identifier may be received at a firstdevice. The first device may be, for example, the playback device 120such as a set-top box. The audio data may be received at a seconddevice. The second device may be a user device (e.g., user device 102)such as a voice activated device. The audio data may correspond to audiogenerated as a result of playback of a content asset by the first deviceor any other device capable of presenting audio or video content to auser. The second device may be configured to send the request for theone or more identifiers in response to receipt at the second device of avoice command comprising one or more keywords. The one or more keywordsmay be stored by the second device in a keyword storage module of thesecond device which may comprise a number of keywords that arerecognizable by the speech processor using, for example, patternmatching, and may allow a user to interact with content presented duringplayback of a content asset. For example, the keyword storage module maystore a plurality of commonly spoken words for interacting with thecontent, such as “order,” “play,” and “record.”

At step 804, it may be determined that the audio data comprises one ormore identifiers. The one or more identifiers may comprise, for example,at least one of an SCTE-35 packet, a unique program identifier (UID), acontent identifier, or an audio watermark. The one or more identifiersmay correspond, for example, to a particular content asset or one ormore products presented during playback of the content asset. At leastone of the first device and the second device may be configured to storea plurality of identifiers associated with a given content asset. Forexample, the first device may be configured to receive the identifiersfrom the service provider prior to or during playback of the contentasset. Additionally or alternatively, at least one of the first deviceor the second device may be configured to determine the one or moreidentifiers generated as a result of playback of the content asset, forexample, using speech processing or an existing transcription of thecontent asset.

At step 806, the one or more identifiers may be sent to the seconddevice. The second device, in response to receiving the one or moreidentifiers, may be configured to cause execution of an operationassociated with the audio data based on a voice command received at thesecond device and the one or more identifiers. In an example where thereceived identifier corresponds to a content asset capable of beingpresented by the first device, the second device may send to the firstdevice an instruction to record or playback the content assetcorresponding to the identifier. In an example where the receivedidentifier corresponds to a product presented during playback of thecontent asset, the second device may place an order for the productthrough an online shopping account associated with the user.

FIG. 9 depicts a computing device that may be used in various aspects,such as the servers, modules, and/or devices depicted in FIGS. 1, 3, 5and 6 . With regard to the example architecture of FIG. 1 , the voiceactivated device 102, computing device 110, and/or the playback device120 may each be implemented in an instance of a computing device 900 ofFIG. 9 . The computer architecture shown in FIG. 9 illustrates aconventional server computer, workstation, desktop computer, laptop,tablet, network appliance, PDA, e-reader, digital cellular phone, orother computing node, and may be utilized to execute any aspects of thecomputers described herein, such as to implement the methods describedin relation to FIGS. 1, 5, 5 and 6 .

The computing device 900 may include a baseboard, or “motherboard,”which is a printed circuit board to which a multitude of components ordevices may be connected by way of a system bus or other electricalcommunication paths. One or more central processing units (CPUs) 904 mayoperate in conjunction with a chipset 906. The CPU(s) 904 may bestandard programmable processors that perform arithmetic and logicaloperations necessary for the operation of the computing device 900.

The CPU(s) 904 may perform the necessary operations by transitioningfrom one discrete physical state to the next through the manipulation ofswitching elements that differentiate between and change these states.Switching elements may generally include electronic circuits thatmaintain one of two binary states, such as flip-flops, and electroniccircuits that provide an output state based on the logical combinationof the states of one or more other switching elements, such as logicgates. These basic switching elements may be combined to create morecomplex logic circuits including registers, adders-subtractors,arithmetic logic units, floating-point units, and the like.

The CPU(s) 904 may be augmented with or replaced by other processingunits, such as GPU(s) 905. The GPU(s) 905 may comprise processing unitsspecialized for but not necessarily limited to highly parallelcomputations, such as graphics and other visualization-relatedprocessing.

A chipset 906 may provide an interface between the CPU(s) 904 and theremainder of the components and devices on the baseboard. The chipset906 may provide an interface to a random access memory (RAM) 908 used asthe main memory in the computing device 900. The chipset 906 may providean interface to a computer-readable storage medium, such as a read-onlymemory (ROM) 820 or non-volatile RAM (NVRAM) (not shown), for storingbasic routines that may help to start up the computing device 900 and totransfer information between the various components and devices. ROM 820or NVRAM may also store other software components necessary for theoperation of the computing device 900 in accordance with the aspectsdescribed herein.

The computing device 900 may operate in a networked environment usinglogical connections to remote computing nodes and computer systemsthrough local area network (LAN) 916. The chipset 906 may includefunctionality for providing network connectivity through a networkinterface controller (NIC) 922, such as a gigabit Ethernet adapter. ANIC 922 may be capable of connecting the computing device 900 to othercomputing nodes over a network 916. It should be appreciated thatmultiple NICs 922 may be present in the computing device 900, connectingthe computing device to other types of networks and remote computersystems.

The computing device 900 may be connected to a mass storage device 926that provides non-volatile storage for the computer. The mass storagedevice 926 may store system programs, application programs, otherprogram modules, and data, which have been described in greater detailherein. The mass storage device 926 may be connected to the computingdevice 900 through a storage controller 924 connected to the chipset906. The mass storage device 926 may consist of one or more physicalstorage units. A storage controller 924 may interface with the physicalstorage units through a serial attached SCSI (SAS) interface, a serialadvanced technology attachment (SATA) interface, a fiber channel (FC)interface, or other type of interface for physically connecting andtransferring data between computers and physical storage units.

The computing device 900 may store data on a mass storage device 926 bytransforming the physical state of the physical storage units to reflectthe information being stored. The specific transformation of a physicalstate may depend on various factors and on different implementations ofthis description. Examples of such factors may include, but are notlimited to, the technology used to implement the physical storage unitsand whether the mass storage device 926 is characterized as primary orsecondary storage and the like.

For example, the computing device 900 may store information to the massstorage device 926 by issuing instructions through a storage controller924 to alter the magnetic characteristics of a particular locationwithin a magnetic disk drive unit, the reflective or refractivecharacteristics of a particular location in an optical storage unit, orthe electrical characteristics of a particular capacitor, transistor, orother discrete component in a solid-state storage unit. Othertransformations of physical media are possible without departing fromthe scope and spirit of the present description, with the foregoingexamples provided only to facilitate this description. The computingdevice 900 may read information from the mass storage device 926 bydetecting the physical states or characteristics of one or moreparticular locations within the physical storage units.

In addition to the mass storage device 926 described herein, thecomputing device 900 may have access to other computer-readable storagemedia to store and retrieve information, such as program modules, datastructures, or other data. It should be appreciated by those skilled inthe art that computer-readable storage media may be any available mediathat provides for the storage of non-transitory data and that may beaccessed by the computing device 900.

By way of example and not limitation, computer-readable storage mediamay include volatile and non-volatile, transitory computer-readablestorage media and non-transitory computer-readable storage media, andremovable and non-removable media implemented in any method ortechnology. Computer-readable storage media includes, but is not limitedto, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasableprogrammable ROM (“EEPROM”), flash memory or other solid-state memorytechnology, compact disc ROM (“CD-ROM”), digital versatile disk (“DVD”),high definition DVD (“HD-DVD”), BLU-RAY, or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage, other magneticstorage devices, or any other medium that may be used to store thedesired information in a non-transitory fashion.

A mass storage device, such as the mass storage device 926 depicted inFIG. 9 , may store an operating system utilized to control the operationof the computing device 900. The operating system may comprise a versionof the LINUX operating system. The operating system may comprise aversion of the WINDOWS SERVER operating system from the MICROSOFTCorporation. According to additional aspects, the operating system maycomprise a version of the UNIX operating system. Various mobile phoneoperating systems, such as IOS and ANDROID, may also be utilized. Itshould be appreciated that other operating systems may also be utilized.The mass storage device 926 may store other system or applicationprograms and data utilized by the computing device 900.

The mass storage device 926 or other computer-readable storage media mayalso be encoded with computer-executable instructions, which, whenloaded into the computing device 900, transforms the computing devicefrom a general-purpose computing system into a special-purpose computercapable of implementing the aspects described herein. Thesecomputer-executable instructions transform the computing device 900 byspecifying how the CPU(s) 904 transition between states, as describedherein. The computing device 900 may have access to computer-readablestorage media storing computer-executable instructions, which, whenexecuted by the computing device 900, may perform the methods describedin relation to FIGS. 1, 3, 5 and 6 .

A computing device, such as the computing device 900 depicted in FIG. 9, may also include an input/output controller 932 for receiving andprocessing input from a number of input devices, such as a keyboard, amouse, a touchpad, a touch screen, an electronic stylus, or other typeof input device. Similarly, an input/output controller 932 may provideoutput to a display, such as a computer monitor, a flat-panel display, adigital projector, a printer, a plotter, or other type of output device.It will be appreciated that the computing device 900 may not include allof the components shown in FIG. 9 , may include other components thatare not explicitly shown in FIG. 9 , or may utilize an architecturecompletely different than that shown in FIG. 9 .

As described herein, a computing device may be a physical computingdevice, such as the computing device 900 of FIG. 9 . A computing nodemay also include a virtual machine host process and one or more virtualmachine instances. Computer-executable instructions may be executed bythe physical hardware of a computing device indirectly throughinterpretation and/or execution of instructions stored and executed inthe context of a virtual machine.

It is to be understood that the methods and systems are not limited tospecific methods, specific components, or to particular implementations.It is also to be understood that the terminology used herein is for thepurpose of describing particular embodiments only and is not intended tobe limiting.

As used in the specification and the appended claims, the singular forms“a,” “an,” and “the” include plural referents unless the context clearlydictates otherwise. Ranges may be expressed herein as from “about” oneparticular value, and/or to “about” another particular value. When sucha range is expressed, another embodiment includes from the oneparticular value and/or to the other particular value. Similarly, whenvalues are expressed as approximations, by use of the antecedent“about,” it will be understood that the particular value forms anotherembodiment. It will be further understood that the endpoints of each ofthe ranges are significant both in relation to the other endpoint, andindependently of the other endpoint.

“Optional” or “optionally” means that the subsequently described eventor circumstance may or may not occur, and that the description includesinstances where said event or circumstance occurs and instances where itdoes not.

Throughout the description and claims of this specification, the word“comprise” and variations of the word, such as “comprising” and“comprises,” means “including but not limited to,” and is not intendedto exclude, for example, other components, integers or steps.“Exemplary” means “an example of” and is not intended to convey anindication of a preferred or ideal embodiment. “Such as” is not used ina restrictive sense, but for explanatory purposes.

Components are described that may be used to perform the describedmethods and systems. When combinations, subsets, interactions, groups,etc., of these components are described, it is understood that whilespecific references to each of the various individual and collectivecombinations and permutations of these may not be explicitly described,each is specifically contemplated and described herein, for all methodsand systems. This applies to all aspects of this application including,but not limited to, operations in described methods. Thus, if there area variety of additional operations that may be performed it isunderstood that each of these additional operations may be performedwith any specific embodiment or combination of embodiments of thedescribed methods.

The present methods and systems may be understood more readily byreference to the following detailed description of preferred embodimentsand the examples included therein and to the Figures and theirdescriptions.

As will be appreciated by one skilled in the art, the methods andsystems may take the form of an entirely hardware embodiment, anentirely software embodiment, or an embodiment combining software andhardware aspects. Furthermore, the methods and systems may take the formof a computer program product on a computer-readable storage mediumhaving computer-readable program instructions (e.g., computer software)embodied in the storage medium. More particularly, the present methodsand systems may take the form of web-implemented computer software. Anysuitable computer-readable storage medium may be utilized including harddisks, CD-ROMs, optical storage devices, or magnetic storage devices.

Embodiments of the methods and systems are described below withreference to block diagrams and flowchart illustrations of methods,systems, apparatuses and computer program products. It will beunderstood that each block of the block diagrams and flowchartillustrations, and combinations of blocks in the block diagrams andflowchart illustrations, respectively, may be implemented by computerprogram instructions. These computer program instructions may be loadedon a general-purpose computer, special-purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions which execute on the computer or other programmabledata processing apparatus create a means for implementing the functionsspecified in the flowchart block or blocks.

These computer program instructions may also be stored in acomputer-readable memory that may direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including computer-readableinstructions for implementing the function specified in the flowchartblock or blocks. The computer program instructions may also be loadedonto a computer or other programmable data processing apparatus to causea series of operational steps to be performed on the computer or otherprogrammable apparatus to produce a computer-implemented process suchthat the instructions that execute on the computer or other programmableapparatus provide steps for implementing the functions specified in theflowchart block or blocks.

The various features and processes described herein may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and sub-combinations are intended to fall withinthe scope of this disclosure. In addition, certain methods or processblocks may be omitted in some implementations. The methods and processesdescribed herein are also not limited to any particular sequence, andthe blocks or states relating thereto may be performed in othersequences that are appropriate. For example, described blocks or statesmay be performed in an order other than that specifically described, ormultiple blocks or states may be combined in a single block or state.The example blocks or states may be performed in serial, in parallel, orin some other manner. Blocks or states may be added to or removed fromthe described example embodiments. The example systems and componentsdescribed herein may be configured differently than described. Forexample, elements may be added to, removed from, or rearranged comparedto the described example embodiments.

It will also be appreciated that various items are illustrated as beingstored in memory or on storage while being used, and that these items orportions thereof may be transferred between memory and other storagedevices for purposes of memory management and data integrity.Alternatively, in other embodiments, some or all of the software modulesand/or systems may execute in memory on another device and communicatewith the illustrated computing systems via inter-computer communication.Furthermore, in some embodiments, some or all of the systems and/ormodules may be implemented or provided in other ways, such as at leastpartially in firmware and/or hardware, including, but not limited to,one or more application-specific integrated circuits (“ASICs”), standardintegrated circuits, controllers (e.g., by executing appropriateinstructions, and including microcontrollers and/or embeddedcontrollers), field-programmable gate arrays (“FPGAs”), complexprogrammable logic devices (“CPLDs”), etc. Some or all of the modules,systems, and data structures may also be stored (e.g., as softwareinstructions or structured data) on a computer-readable medium, such asa hard disk, a memory, a network, or a portable media article to be readby an appropriate device or via an appropriate connection. The systems,modules, and data structures may also be transmitted as generated datasignals (e.g., as part of a carrier wave or other analog or digitalpropagated signal) on a variety of computer-readable transmission media,including wireless-based and wired/cable-based media, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). Suchcomputer program products may also take other forms in otherembodiments. Accordingly, the present invention may be practiced withother computer system configurations.

While the methods and systems have been described in connection withpreferred embodiments and specific examples, it is not intended that thescope be limited to the particular embodiments set forth, as theembodiments herein are intended in all respects to be illustrativerather than restrictive.

Unless otherwise expressly stated, it is in no way intended that anymethod set forth herein be construed as requiring that its operations beperformed in a specific order. Accordingly, where a method claim doesnot actually recite an order to be followed by its operations or it isnot otherwise specifically stated in the claims or descriptions that theoperations are to be limited to a specific order, it is no way intendedthat an order be inferred, in any respect. This holds for any possiblenon-express basis for interpretation, including: matters of logic withrespect to arrangement of steps or operational flow; plain meaningderived from grammatical organization or punctuation; and the number ortype of embodiments described in the specification.

It will be apparent to those skilled in the art that variousmodifications and variations may be made without departing from thescope or spirit of the present disclosure. Other embodiments will beapparent to those skilled in the art from consideration of thespecification and practices described herein. It is intended that thespecification and example figures be considered as exemplary only, witha true scope and spirit being indicated by the following claims.

What is claimed is:
 1. A method comprising: receiving, by a user deviceassociated with a premises, audio comprising a voice command; sending,based on determining context information associated with the premises isneeded, a request for the context information; receiving the contextinformation, wherein the context information comprises one or moreidentifiers associated with one or more premises devices of a pluralityof premises devices located at the premises; and causing, based on thevoice command and the one or more identifiers, an operation to beperformed by the one or more premises devices.
 2. The method of claim 1,wherein the voice command comprises one or more keywords.
 3. The methodof claim 1, wherein the determining context information associated withthe premises is needed comprises: determining the voice command isassociated with an operation associated with the one or more premisesdevices of the plurality of premises devices; and determining that oneor more identifiers associated with the one or more premises devices ofthe plurality of premises devices is needed to cause the operation to beperformed.
 4. The method of claim 1, wherein the one or more identifiersindicate which of the one or more premises devices of the plurality ofpremises devices are in condition to perform the operation.
 5. Themethod of claim 1, wherein the user device comprises a voice activateddevice.
 6. The method of claim 1, wherein the one or more identifiersare stored for a period of time.
 7. The method of claim 1, wherein thesending the request comprises sending the request to a computing devicelocated external to the premises, and wherein the receiving the contextinformation comprising receiving the context information from thecomputing device.
 8. The method of claim 1, wherein the one or morepremises devices of the plurality of premises devices comprises at leastone of: a light, an appliance, a thermostat, or a smart device.
 9. Anon-transitory computer readable medium storing instructions that, whereexecuted, cause: receiving, by a user device associated with a premises,audio comprising a voice command; sending, based on determining contextinformation associated with the premises is needed, a request for thecontext information; receiving the context information, wherein thecontext information comprises one or more identifiers associated withone or more premises devices of a plurality of premises devices locatedat the premises; and causing, based on the voice command and the one ormore identifiers, an operation to be performed by the one or morepremises devices.
 10. The non-transitory computer readable medium ofclaim 9, wherein the voice command comprises one or more keywords. 11.The non-transitory computer readable medium of claim 9, wherein theinstructions that cause determining context information associated withthe premises is needed cause: determining the voice command isassociated with an operation associated with the one or more premisesdevices of the plurality of premises devices; and determining that oneor more identifiers associated with the one or more premises devices ofthe plurality of premises devices is needed to cause the operation to beperformed.
 12. The non-transitory computer readable medium of claim 9,wherein the one or more identifiers indicate which of the one or morepremises devices of the plurality of premises devices are in conditionto perform the operation.
 13. The non-transitory computer readablemedium of claim 9, wherein the user device comprises a voice activateddevice.
 14. The non-transitory computer readable medium of claim 9,wherein the one or more identifiers are stored for a period of time. 15.The non-transitory computer readable medium of claim 9, wherein thesending the request comprises sending the request to a computing devicelocated external to the premises, and wherein the receiving the contextinformation comprising receiving the context information from thecomputing device.
 16. The non-transitory computer readable medium ofclaim 9, wherein the one or more premises devices of the plurality ofpremises devices comprises at least one of: a light, an appliance, athermostat, or a smart device.
 17. A system comprising: a plurality ofpremises devices associated with a premises; a user device associatedwith the premises configured to: receive, by a user device associatedwith a premises, audio comprising a voice command; send, based ondetermining context information associated with the premises is needed,a request for the context information; receive the context information,wherein the context information comprises one or more identifiersassociated with one or more premises devices of the plurality ofpremises devices located at the premises; and cause, based on the voicecommand and the one or more identifiers, an operation to be performed bythe one or more premises devices.
 18. The system of claim 17, whereinthe voice command comprises one or more keywords.
 19. The system ofclaim 17, wherein the user device is configured to determine contextinformation associated with the premises is needed by: determining thevoice command is associated with an operation associated with the one ormore premises devices of the plurality of premises devices; anddetermining that one or more identifiers associated with the one or morepremises devices of the plurality of premises devices is needed to causethe operation to be performed.
 20. The system of claim 17, wherein theone or more identifiers indicate which of the one or more premisesdevices of the plurality of premises devices are in condition to performthe operation.
 21. The system of claim 17, wherein the user devicecomprises a voice activated device.
 22. The system of claim 17, whereinthe one or more identifiers are stored for a period of time.
 23. Thesystem of claim 17, wherein the user device is configured to: send therequest by sending the request to a computing device located external tothe premises, and receive the context information by receiving thecontext information from the computing device.
 24. The system of claim17, wherein the one or more premises devices of the plurality ofpremises devices comprises at least one of: a light, an appliance, athermostat, or a smart device.